System and method for bandwidth reduction and communication of visual events

ABSTRACT

This invention provides a system and method that employs reduction in bandwidth and the amount of data stored, along with queuing of data, so that the significant and/or relevant shipboard visual information is detected and communicated continuously from the ship (or other remote vehicle/location) to shore without loss of useful/high-priority information and within the available bandwidth of that typical, available satellite link. The system and method supports remote configuration and management from shore to the ship over the same communications channel but in the reverse direction. The system and method further facilitates the logical summarization of continuous visibility into shipboard activities, shipboard behavior, and shipboard status so that a land/shore-based manager/reviewer/auditor can review, comprehend and synthesize such information, at a glance, using a land-based user interface/dashboard, related to events that have recently transpired on the sailing commercial merchant vessel without the need to review hours/days/weeks of multiple channels of video.

FIELD OF THE INVENTION

This application relates to systems and methods for communicating visualdata and related events over a reduced bandwidth communication link, anduser interfaces for viewing and manipulating such data and events.

BACKGROUND OF THE INVENTION

International shipping is a critical part of the world economy.Ocean-going, merchant freight vessels are employed to carry virtuallyall goods and materials between ports and nations. The current approachto goods shipments employs intermodal cargo containers, which are loadedand unloaded from the deck of ships, and are carried in a stackedconfiguration. Freight is also shipped in bulk carriers (e.g. grain) orliquid tankers (e.g. oil). The operation of merchant vessels can behazardous and safety concerns are always present. Likewise, passengervessels, with the precious human cargo are equally, if not more,concerned with safety of operations and adherences to rules andregulations by crew and passengers. Knowledge of the current status ofthe vessel, crew and cargo can be highly useful in ensuring safe andefficient operation.

In many areas of commercial and/or government activity, visualmonitoring (manual and automated surveillance), and other status sensorsis employed to ensure safe and rule-conforming operation. Theseapproaches, however, entail the generation and transmission of largevolumes of data to a local or remote location, where such data is storedand/or analyzed by management personnel. However, unlike most land-based(i.e. wired, fiber or high-bandwidth wireless) communication links, itis often much more challenging to transmit useful data (e.g. visualinformation) from ship-to-shore. It can be assumed that ten (10)channels of raw video data are the minimum number needed to provideshipboard visibility and a 5 Mb/s of uplink speed per HD video channelor 50 Mb/s in aggregate is required. Conversely a typical satellite linkto/from a ship is 1/200 or 1/400 of that size, and transmits only 256Kb/s or 128 Kb/s.

It is desirable to provide a system and method that enables continuousvisibility into the shipboard activities, shipboard behavior, andshipboard status of an at-sea commercial merchant vessel (cargo,fishing, industrial, and passenger). It is further desirable that thatthe transmitted visual data and associated status be accessible via aninterface that aids users in manipulating, organizing and acting uponsuch information.

SUMMARY OF THE INVENTION

This invention overcomes the disadvantages of the prior art by providinga system and method that employs reduction in bandwidth and the amountof data stored bandwidth, along with queuing of data so that thesignificant and/or relevant shipboard visual information is detected andcommunicated continuously from the ship (or other remotevehicle/location) to shore without (free of) the loss ofuseful/high-priority information and within the available bandwidth ofthat typical, available satellite link. The system and method supportsremote configuration and management from shore to the ship over the samecommunications channel but in the reverse direction. The system andmethod further facilitates the logical summarization of continuousvisibility into shipboard activities, shipboard behavior, and shipboardstatus so that a land/shore-based manager/reviewer/auditor can review,comprehend and synthesize such information, at a glance, using aland-based computer graphical user interface (GUI), which can be definedby a dashboard, related to events that have recently transpired on thesailing commercial merchant vessel without the need to reviewhours/days/weeks of (e.g.) 10+ channels of video. Notably, the systemand method effectively provides that visibility to the land-basedcomputer dashboard.

In an illustrative embodiment, a system and method for storage andbandwidth reduction of visual events in association with an activity isprovided. At least one visual event detector is provided, and caninclude at least one camera that acquires images of the activity. Atleast one processor receives the images and a visual event detectionprocess operates at least in part on the processor that processes theimages. A queuing process identifies visual events from the at least onevisual event detector and stores the visual events. A communicationsprocess enables a reduced-bandwidth communications channel adapted totransmit the stored visual events in a reduced-bandwidth and queuedformat to a remote location. Illustratively, the visual events cancomprise activities related to at least one of bridge operations, cargooperations, maintenance operations, safety operations, and vetting andsecurity with respect to a vessel. The visual events can be based uponsoftware-based and hardware-based event detectors that identifypredetermined characteristics within one or more of the images anddetermine a state based upon the characteristics. The visual events canprovide data primitives that are derived into user information, andthese data primitives can, thus, include at least one of statistics,pattern matches, metrics and time. The stored visual events can becommunicated across the communications channel based upon user-setpriorities, wherein higher priority events are communicated first in aqueue from the stored visual events. A user interface can displayinformation of the at least one of the bridge operations, cargooperations, maintenance operations, safety operations, and vetting andsecurity at a base location in one or more predetermined presentationformats. The user interface can include inputs that enable the user-setpriorities. The user-interface can include inputs that enable adding orsetting configuration parameters for the event detectors. The userinformation can also include at least one of fleet reports on aplurality of vessels associated with the user interface, vessel reports,occurrence of discrete events and occurrence of complex eventscomprising a plurality of discrete events. The information can bedisplayed in a dashboard format comprising one or more visual windows,alerts and numerical data.

In an illustrative embodiment, a user interface is provided, which islocated at a base location and receives visual events over acommunication channel from a remote, moving or stationary object, inwhich the object includes at least one visual event detector having atleast one camera that acquires images of the activity. At least oneprocessor is provided, which receives the images, and a visual eventdetection process operates, at least in part, on the processor toprocess the images. A communications process enables a communicationschannel adapted to transmit the stored visual events in a queued formatto the base location. The user interface includes a dashboard fordisplaying information associated with the object. The user interfaceenables configurations of at least one of (a) priority of visual eventsto be communicated over the communications channel and (b) configurationof characteristics associated with the visual events. The communicationschannel can be a reduced-bandwidth, wireless communications channel. Theobject can be a sea-going vessel and the visual events can relate to atleast one of safety, maintenance, cargo handling, bridge operations,vetting and security relative to the vessel. A dashboard display on theuser interface can provide at least one of configuration, status andperformance information relative to the vessel. The dashboard display onthe user interface can provide at least one of configuration, status andperformance information relative to each vessel of a fleet of vessels.Illustratively, the information includes one or more forms of displayedsensor data, having at least one of a label, highlight, comment andannotation visually appended thereto. The information handled and/ordisplayed by the interface can include a workflow provided between oneor more users or vessels. The communication channel can be arranged totransmit control data from the user interface to one or more of thevessels and receive feedback via the user interface relative to actionson the vessel with respect to the control data.

In an illustrative embodiment, a method for storage and bandwidthreduction of visual events in association with an activity is provided.The method acquires images of the activity with at least one camera andperforming visual event detection with a processor. Visual events fromthe at least one visual event detector are identified and stored in aqueue based upon predetermined parameters. Communication with a remotelocation occurs over a reduced-bandwidth communications channel adaptedto transmit the stored visual events in a reduced-bandwidth and queuedformat to the remote location.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention description below refers to the accompanying drawings, ofwhich:

FIG. 1 is a diagram showing an overview of a system and associatedmethod for acquiring, transmitting analyzing and reporting visual andother sensor information with respect to a communication link typicallycharacterized by reduced bandwidth according to an illustrativeembodiment;

FIG. 1A is a block diagram showing data and operations employed by thesystem and method of FIG. 1;

FIG. 1B is a diagram showing acquisition of images and other data forexpected event detection according to the system and method of FIG. 1;

FIG. 2 is a flow diagram showing detection and reporting of visualevents and associated data by the processors and processes of the systemand method of FIG. 1;

FIG. 3 is a flow diagram showing the detection of visual events usingprocessors and processes of the system and method of FIG. 1 in theexample of a bridge routine on a sea-going merchant vessel;

FIG. 4 is a flow diagram showing the detection of visual events usingprocessors and processes of the system and method of FIG. 1 in theexample of the performance of safety rounds by personnel (crew) on asea-going merchant vessel;

FIG. 5 is a flow diagram showing the detection of visual events usingprocessors and processes of the system and method of FIG. 1 in theexample of performing activities with respect to cargo handling on asea-going merchant vessel;

FIG. 6 is a diagram showing an exemplary graphical user interface (GUI)displayed by a processor and associated process in the system and methodof FIG. 1, depicting an exemplary dashboard of vessel fleet activity andstatus;

FIG. 7 is a diagram showing an exemplary GUI displayed by a processorand associated process in the system and method of FIG. 1, depicting anexemplary dashboard of vessel activity, including current status andperformance;

FIG. 8 is a diagram showing an exemplary GUI displayed by a processorand associated process in the system and method of FIG. 1, depicting ananother exemplary dashboard of vessel activity, including current statusand performance;

FIG. 9 is a diagram showing an exemplary GUI displayed by a processorand associated process in the system and method of FIG. 1, depicting anexemplary dashboard for cargo procedure tracking on a vessel;

FIG. 10 is a diagram showing an exemplary GUI displayed by a processorand associated process in the system and method of FIG. 1, depicting anexemplary dashboard for maintenance activity tracking on a vessel;

FIG. 11 is a diagram showing an exemplary GUI displayed by a processorand associated process in the system and method of FIG. 1, depicting anexemplary dashboard for bridge activity/behavior tracking on a vessel;

FIG. 12 is a diagram showing an exemplary GUI displayed by a processorand associated process in the system and method of FIG. 1, depicting anexemplary dashboard for safety status and procedure tracking on avessel;

FIG. 13 is a diagram showing an exemplary GUI displayed by a processorand associated process in the system and method of FIG. 1, depicting anexemplary dashboard for cargo procedure reporting on a vessel;

FIG. 14 is a diagram showing an exemplary GUI displayed by a processorand associated process in the system and method of FIG. 1, depicting anexemplary dashboard for maintenance activity reporting on a vessel;

FIG. 15 is a diagram showing an exemplary GUI displayed by a processorand associated process in the system and method of FIG. 1, depicting anexemplary dashboard for bridge activity/behavior reporting on a vessel;and

FIG. 16 is a diagram showing an exemplary GUI displayed by a processorand associated process in the system and method of FIG. 1, depicting anexemplary dashboard for safety status and procedure reporting on avessel.

DETAILED DESCRIPTION I. System Overview

FIGS. 1 and 1A show an arrangement 100 for tracking and reporting uponvisual, and other, events generated by visual sensors aboard ship thatcreate video data streams, visual detection of events aboard ship basedon those video data streams, aggregation of those visual detectionsaboard ship, prioritization and queuing of the aggregated detectionsinto events, bandwidth reduction of the video data streams incombination with the aggregated events, sending the events over thereduced bandwidth communications channel to shore, reporting the eventsto a user-interface on shore, and further aggregation of the events frommultiple ships and multiple time periods into a fleet-wide aggregationthat can present information over time. The system and method hereinfurther provides the ability to configure and setup the system describedabove to select or not select events for presentation in order to reduceconfusion for the person viewing the dashboard as well as to set thepriority for communicating particular events or classes of events overthe reduced bandwidth communications channel so that the most importantevents are communicated at the expense of less important events.

FIG. 1, the arrangement 100 particularly depicts a shipboard location110 includes a camera (visual sensor) array 112 comprising a pluralityof discrete cameras 118 (and/or other appropriateenvironmental/event-driven sensors) that are connected to wired and/orwireless communication links (e.g. that are part of a TCP/IP LAN orother protocol-driven data transmission network 116) via one or moreswitches, routers, etc. 114. Image (and other) data from the (camera)sensors 118 is transmitted via the network 116. Note that cameras canprovide analog or other format image data to a remote receiver thatgenerates digitized data packets for use of the network 116. The cameras118 can comprise conventional machine vision cameras or sensorsoperating to collect raw video or digital image data, which can be basedupon two-dimensional (2D) and/or three-dimensional (3D) imaging.Furthermore, the image information can be grayscale (monochrome), color,and/or near-visible (e.g. infrared (IR)). Likewise, other forms ofevent-based cameras can be employed.

Note that data used herein can include both direct feeds fromappropriate sensors and also data feeds from other data sources that canaggregate various information, telemetry, etc. For example, locationand/or directional information can be obtained from navigation systems(GPS etc.) or other systems (e.g. via APIs) through associated dataprocessing devices (e.g. computers) that are networked with a server 130for the system. Similarly, crew members can input information via anappropriate user interface. The interface can request specificinputs—for example logging into or out of a shift, providing healthinformation, etc.—or the interface can search for information that isotherwise input by crew during their normal operations—for example,determining when a crew member is entering data in the normal course ofshipboard operations to ensure proper procedures are being attended toin a timely manner.

The shipboard location 110 can further include a local image/other datarecorder 120. The recorder can be a standalone unit, or part of abroader computer server arrangement 130 with appropriate processor(s),data storage and network interfaces. The server 130 can performgeneralized shipboard, or dedicated, to operations of the system andmethod herein with appropriate software. The server 130 communicateswith a work station or other computing device 132 that can include anappropriate display (e.g. a touchscreen) 134 and other components thatprovide a graphical user interface (GUI). The GUI provides a user onboard the vessel with a local dashboard for viewing and controllingmanipulation of event data generated by the sensors 118 as describedfurther below. Note that display and manipulation of data can include,but is not limited to enrichment of the displayed data (e.g. images,video, etc.) with labels, comments, flags, highlights, and the like.

The information handled and/or displayed by the interface can include aworkflow provided between one or more users or vessels. Such a workflowwould be a business process where information is transferred from userto user (at shore or at sea interacting with the application over theGUI) for action according to the business procedures/rules/policies.This workflow automation is commonly referred to as “robotic processautomation.”

The processes 150 that run the dashboard and other data-handlingoperations in the system and method can be performed in whole or in partwith the onboard server 130, and/or using a remote computing (server)platform 140 that is part of a land-based, or other generally fixed,location with sufficient computing/bandwidth resources (a base location142). The processes can generally include 150 a computation process 152that handles sensor data to meaningful events. This can include machinevision algorithms and similar procedures. A data-handling process 154can be used to derive events and associated status based upon theevents—for example movements of the crew and equipment, cargo handling,etc. An information process 156 can be used to drive dashboards for oneor more vessels and provide both status and manipulation of data for auser on the ship and at the base location.

Data is communicated between the ship (or other remote location) 110 andthe base 142 occurs over one or more reduced-bandwidth wirelesschannels, which can be facilitated by a satellite uplink/downlink 160,or another transmission modality—for example, long-wavelength, over-airtransmission. Moreover, other forms of wireless communication can beemployed such as mesh networks and/or underwater communication (forexample long-range, sound-based communication and/or VLF). Note thatwhen the ship is located near a land-based high-bandwidth channel orphysically connected by-wire while at port, the system and method hereincan be adapted to utilize that high-bandwidth channel to send allpreviously unsent low-priority events, alerts, and/or image-basedinformation.

The (shore) base server environment 140 communicates via an appropriate,secure and/or encrypted link (e.g. a LAN or WAN (Internet)) 162 with auser workstation 170 that can comprise a computing device with anappropriate GUI arrangement, which defines a user dashboard 172 allowingfor monitoring and manipulation of one or more vessels in a fleet overwhich the user is responsible and manages.

Referring further to FIG. 1A, the data handled by the system is shown infurther detail. The data acquired aboard the vessel environment 110, andprovided to the server 130 can include a plurality of possible, detectedvisual (and other sensor-based) events. These events can be generated byaction of software and/or hardware based detectors that analyze visualimages and/or time-sequences of images acquired by the cameras. Withfurther reference to FIG. 1B, visual detection is facilitated by aplurality of 2D and/or 3D camera assemblies depicted as cameras 180 and182 using ambient or secondary sources of illumination 183 (visibleand/or IR). The camera assemblies image scenes 184 located on board(e.g.) a ship. The scenes can relate to, among other subjects, maritimeevents, personnel safety and/or cargo. The images are directed as imagedata to the event detection server or processor 186 that also receivesinputs from a plan or program 187 that characterizes events and eventdetection and a clock 188 that establishes a timeline and timestamp forreceived images. The event detection server or processor 186 can alsoreceive inputs from a GPS receiver 189 to stamp the position of the shipat the time of the event and can also receive input from anarchitectural plan 190 of the vessel (that maps onboard locations onvarious decks) to stamp the position of the sensor within the vesselthat sent the input. The event server/processor 186 can comprise one ormore types and/or architectures of processor(s), including, but notlimited to, a central processing unit (CPU—for example one or moreprocessing cores and associated computation units), a graphicalprocessing unit (GPU—operating on a SIMD or similar arrangement), tensorprocessing unit (TPU) and/or field programmable gate array (FPGA—havinga generalized or customized architecture).

Referring again to FIG. 1A, the base location dashboard 172 isestablished on a per-ship and/or per fleet basis and communicates withthe shipboard server 130 over the communications link 160 in a mannerthat is typically reduced in bandwidth, and possibly intermittent inperforming data transfer operations. The link 160 transmits events andstatus updates 162 from the shipboard server 130 to the dashboard 172and event priorities, camera settings and vision system parameters 164from the dashboard 172 to the shipboard server. More particularly, thedashboard displays and allows manipulation of events reports and logs173, alarm reports and logs 174, priorities for events, etc. 175, camerasetup 176 and vision system task selection and setup relevant to eventdetection, etc. 177. The shipboard server 130 includes variousfunctional modules, including visual event bandwidth reduction 132 thatfacilitates transmission over the link 160; alarm and status polling andqueuing 133 that determines when alarms or various status items haveoccurred and transmits them in the appropriate priority order; prioritysetting 134 that selects the priorities for reporting and transmission;and a data storage that maintains image and other associated data from apredetermined time period 135.

Note that, in various embodiments, the bandwidth of the communicationslink between vessel and base can be limited by external systems such asQoS-quality of service-settings on routers/link OR by the internalsystem (edge server 130)—for example to limit usage to (e.g.) 15% oftotal available communication bandwidth. This limitation in bandwidthcan be based on a variety of factors, including, but not limited to, thetime of day and/or a communications satellite usage cost schedule. Anappropriate instruction set can be programmed into the server usingconventional or custom control processes. The specific settings for suchbandwidth control can also be directed by the user via the GUI.

II. Visual Detectors

As shown in FIG. 1B, various imaged events are determined from acquiredimage data using appropriate processes/algorithms 188 performed by theprocessor(s) 186. These can include classical algorithms, which are partof a conventional vision system, such as those available from (e.g.)Keyence, Cognex Corporation, MVTec, or HIKVision. Alternatively, theclassical vision system could be based on open source such as OpenCV.Such classical vision systems can include a variety of vision systemtools, including, but not limited to, edge finders, blob analyzers,pattern recognition tools, etc. The processor(s) 186 can also employmachine learning algorithms or deep learning algorithms, which can becustom built or commercially available from a variety of sources, andemploy appropriate deep-learning frameworks such as caffe, tensorflow,torch, keras and/or OpenCV. The network could be a mask R-CNN or Yolov3detector. See also URL addresshttps://engineer.dena.com/posts/2019.05/survey-of-cutting-edge-computer-vision-papers-human-recognition/on the WorldWideWeb.

As shown in FIG. 1A, the visual detectors relate to maritime events 191,ship personnel safety behavior and events 192, maintenance operation andevents 193, ship cargo condition and events related thereto 194, and/ornon-visual alarms, such as smoke, fire, and/or toxic gas detection viaappropriate sensors. By way of non-limiting example, some particulardetected events and associated detectors relate to the following:

(a) A person is present at their station at the expected time andreports the station, start time, end time, and elapsed time;

(b) A person has entered a location at the expected time and reports thelocation, start time, end time, and elapsed time;

(c) A person moved through a location at the expected time and reportsthe location, start time, end time, and elapsed time;

(d) A person is performing an expected activity at the expected locationat the expected time and reports the location, start time, end time, andelapsed time—the activity can include (e.g.) watching, monitoring,installing, hose-connecting or disconnecting, crane operating, tyingwith ropes;

(e) a person is running, slipping, tripping, falling, lying down, usingor not using handrails at a location at the expected time and reportsthe location, start time, end time, and elapsed time;

(f) A person is wearing or not wearing protective equipment whenperforming an expected activity at the expected location at the expectedtime and reports the location, start time, end time, and elapsedtime—protective equipment can include (e.g.) a hard-hat, left or rightglove, left or right shoe/boot, ear protection, safety goggles,life-jacket, gas mask, welding mask, or other protection;

(g) A door is open or closed at a location at the expected time andreports the location, start time, end time, and elapsed time;

(h) An object is present at a location at the expected time and reportsthe location, start time, end time and elapsed time—the object caninclude (e.g.) a gangway, hose, tool, rope, crane, boiler, pump,connector, solid, liquid, small boat and/or other unknown item;

(i) That normal operating activities are being performed using at leastone of engines, cylinders, hose, tool, rope, crane, boiler, and/or pump;and

(j) That required maintenance activities are being performed on engines,cylinders, boilers, cranes, steering mechanisms, HVAC, electrical,pipes/plumbing, and/or other systems.

Note that the above-recited listing of examples (a j) are only some of awide range of possible interactions that can for the basis of detectorsaccording to illustrative embodiments herein. Those of skill shouldunderstand that other detectable events involving person-to-person,person-to-equipment or equipment-to-equipment interaction are expresslycontemplated.

In operation, an expected event visual detector takes as input thedetection result of one or more vision systems aboard the vessel. Theresult could be a detection, no detection, or an anomaly at the time ofthe expected event according to the plan. Multiple events or multipledetections can be combined into a higher-level single events. Forexample, maintenance procedures, cargo activities, or inspection roundsmay result from combining multiple events or multiple detections. Notethat each visual event is associated with a particular (or several)vision system camera(s) 118, 180, 182 at a particular time and theparticular image or video sequence at a known location within thevessel. The associated video can be optionally sent or not sent witheach event or alarm. When the video is sent with the event or alarm, itmay be useful for later validation of the event or alarm. Notably, thediscrete images and/or short-time video frame sequences actuallyrepresent a small fraction of the video stream, and consequentlyrepresent a substantial reduction in the bandwidth required fortransmission in comparison to the entire video sequence over thereduced-bandwidth link. Moreover, in addition to compacting the video byreducing it to a few images or short-time sequence, the system canreduce the images in size either by cropping the images down tosignificant or meaningful image locations required by the detector or byreducing the resolution say from the equivalent of high-definition (HD)resolution to standard-definition (SD) resolution, or below standardresolution.

In addition to reducing bandwidth by identifying events via the visionsystem and cropping such images where appropriate, the number of imageframes can be reduced, in a sequence thereof, by increasing the intervalof time between frames. Moreover, bandwidth can be even further reducedusing the procedures above, and then subjecting (all on the shipboardserver side) the event-centric, cropped, spaced-apart, usingcommercially available or customized lossy or lossless image compressiontechniques. Such techniques can include, but are not limited to discretecosine transform (DCT), run-length encoding (RLE), predictive coding,and/or Lempel-Ziv-Welch (LZW). The images or video sequences NOTassociated with visual events may be stored for some period of time onboard the vessel.

The shipboard server establishes a priority of transmission for theprocessed visual events that is based upon settings provided from auser, typically operating the on-shore (base) dashboard. The shipboardserver buffers these events in a queue in storage that can be orderedbased upon the priority. Priority can be set based on a variety offactors—for example personnel safety and/or ship safety can have firstpriority and maintenance can have last priority, generally mapping tothe urgency of such matters. By way of example, all events in the queuewith highest priority are sent first. They are followed by events withlower priority. If a new event arrives shipboard with higher priority,then that new higher priority event will be sent ahead of lower priorityevents. It is contemplated that the lowest priority events can bedropped if higher priority events take all available bandwidth. Theshipboard server receives acknowledgements from the base server on shoreand confirms that events have been received and acknowledged on shorebefore marking the shipboard events as having been sent. Multiple eventsmay be transmitted prior to receipt (or lack of receipt) ofacknowledgement. Lack of acknowledgement potentially stalls the queue orrequires retransmission of an event prior to transmitting all nextevents in the priority queue on the server. The shore-based serverinterface can configure or select the visual event detectors over thecommunications link. In addition to visual events, the system cantransmit non-visual events like a fire alarm signal or smoke alarmsignal.

III. Detection Flow

As shown in FIG. 2, an operating procedure 200 for generalized detectionflow used in performing the system is shown. The operation can becharacterized in three phases or segments, computation 210, generationof data primitives 220 and information creation 230 and presentation 240to users via the shore-based dashboard. Alternatively, some or all ofthe functions herein can be implemented by users via a ship-baseddashboard, that affects programming on at least one of the local serveror the base server. The shipboard dashboard can also act as a passiveterminal that transmits instructions back to the base interface over thecommunications link so that such instructions can be acted upon throughthe base. The computation phase 210 comprises measurement 212 usingsensors and performing visual detection 213. These generate a set ofmetrics 222 that are displayed to the user as discrete events 232. Thecomputation phase 210 uses event sequencing (priority) 214, filtering(via cropping, compression, etc.), qualification of events 216 basedupon rules 217 to provide pattern matches 224 according to a time seriesof events 226. This data is presented as complex events 234. Thesecomplex events 234 can comprise a scenario, such as a maintenance tasksuccessfully performed, or the occurrence of a safety breach. Thecomputation phase 210 can aggregate visual and other events 218 andderive statistics 228—for example the number of safety breaches over atime interval, etc. These statistics 228 can be presented to theshore-based user as individual vessel reports 236 and fleet reports 238that provide valuable information to the user regarding behavior andperformance at various factors related to the events in aggregate.

FIG. 3 shows a detection flow procedure 300 in the example of bridgeroutines for one or more vessels in a fleet. At the computation phase310, the sample detectors 312 provided by visual and other detectorsinclude (e.g.) a person crossing or stopping at a location, a personinteracting with equipment, a person walking, sitting, not-moving(stationary), a person staring at a location, a person wearing earphonesand/or lights off at the location. In the associated data primitivesgeneration phase 320, sample detected metrics 322 are provided,including (e.g.) starting time and ending time, duration, number ofparticipants, the bridge station visited, a protocol step executed and anon-conformity with protocols. Event samples 324 can include participantname(s) identified as performing the shift, when the shift started,whether a given participant's shift was longer or shorter than normal,missing personnel and/or excess/unauthorized personnel on the bridge. Inthe exemplary information phase sample reports 332 are created that caninclude (e.g.) shift duration over time, shift participation (headcount), equipment interaction time statistics, distribution—for examplenumber of shifts X duration and a location graph (e.g. a heatmap) thatcan be based upon month, week, day, etc. In the information phase 330,the sample reports 332 can be presented as vessel reports 334 and fleetreports. Sample detected metrics 322 and event samples 324 can bepresented to the user as discrete events 338 and complex events 339.

FIG. 4 shows a detection flow procedure 400 in the example of safetyrounds for one or more vessels in a fleet. At the computation phase 410,the sample detectors 412 provided by visual and other detectors include(e.g.) the location of the event, person interacting with equipment,person stopping at a location, person walking or staring at a location,person wearing a hard-hat, life vest or other protective equipmentand/or holding a safety tool, such as a fire extinguisher, flashlight,etc. In the data primitives phase 420 sample detected metrics caninclude (e.g.) starting or ending time of an event, duration, number ofparticipants, station visited protocol step executed and/orround-specific protective equipment (PPE) employed. Event samples 424can include whether a safety round was not performed for a predeterminednumber of hours and a round taking X % more or less time than normal, around performed by X number of personnel, a round started late by Xminutes, a round performed without needed PPE and/or a round completedin X minutes. The information phase 432 provides sample reports 432,based upon events, including duration over time, participation, safetyprotocol compliance, station time requirements, distribution (e.g.number of rounds X duration) and/or a graph/heat map based upon month,day, week, etc. Vessel reports 434 and fleet reports 436. Theinformation phase 430 also reports discrete events 438 and complexevents 439 based upon sample detected events 422 and event samples 424.

FIG. 5 shows a detection flow procedure 500 in the example of cargooperations for one or more vessels in a fleet. At the computation phase510, sample detectors 512 can include a pipe connected, a pipedisconnected, a person interacting with equipment, a person standing,arriving or leaving, a person wearing a hard-hat, gloves, goggles and/orother PPE. The data primitives phase 520 provides sample detectedmetrics 522 include starting and ending time, duration number ofpersonnel participating, a protocol step executed and/or PPE employed inthe task(s). Event samples 524 can include a task complete in X minutes,task completion X % larger or shorter than usual, the task performed byX personnel and/or a task performed without (free of) PPE of X type. Inthe information phase 530 sample reports 532 can include duration overtime, participation, protocol compliance, location/log, distribution(e.g. number of drills X duration) and/or non-conformities versusnormal/standard operation. These can be presented as vessel reports 534or fleet reports 536. Sample detected metrics 522 and event samples 524are reported as discrete events 538 and complex events 539.

Other exemplary detection flows can be provided as appropriate togenerate desired information on activities of interest by the ship'spersonnel and systems. Such detection flows employ relevant detectortypes, parameters, etc.

IV. User Interface Dashboard

A. Status and Performance

FIG. 6 shows a display of an exemplary fleet dashboard 600 that canprovide a user with all key information in a single display screen(which can be a single or multiple physical displays in a consolearrangement). The exemplary display 600 can include a plurality ofwindows and numerical fields that provide the user with an interactiveexperience. For example, an overall fleet status window 610 with alarms(e.g. red circle), cautions (e.g. yellow triangle) and all clearconditions (e.g. green square) is shown. Clicking can allow a drill-downto other specific windows. Similar status windows are provided acrossthe top for cargo procedures 612, maintenance activities 614, bridgebehavior 616 and safety activities 618. A world (or selectable smallerregion) map 620 is provided with flags for vessel locations within thefleet. Locations can be established using GPS and other appropriatelocation technologies. A listing of recent activity 622 is also providedin the center of the display 600. A visual window 630 is shown with aselected event of interest (e.g. a most-recent event). In this case, theevent displays a location on the ship and highlights 632 a person orother relevant activity. A film strip 634 can be provided (e.g. above orbelow the window 630) to allow selection of events—for example in a timesequence. A synopsis of events can be played using the button 636,and/or other play/pause buttons can be used to replay visual events. Ashaded or colored (e.g. heat map) graph can be provided relative to a(e.g.) horizontal time base to show status and/or activities at a fleetlevel for each category—cargo, maintenance, bridge operations andsafety.

FIG. 7 shows a display 700 of vessel status and performance that can beselected from the main fleet dashboard 600, or another mechanism. Theexemplary status box 710 in the upper right of the display 700 allowsselection of the vessel from a menu 702, and contains the selectedvessel name or identifier, along with the respective number of alarms712, warnings 714 and all clear conditions 716. Status conditions forcargo 720, maintenance 722, bridge behavior 724, and safety precautions726. A map 730 shows vessel location. A 3D projection, which can bemanipulated for orientation and zoom, shows event locations withidentifiers 742 related to sensors/cameras, and highlights 744 oflocations with alerts/alarms. Selecting a particular sensor/camera canbring up status data for present and past activity in central listing750. The image display window 760 shows the images and sequencesassociated with the sensor/camera. This can be manipulated usingappropriate buttons. A graph or heat map 760 shows activities and statusfor the different categories relative to the selected vessel.

FIG. 8 shows an additional view of the vessel dashboard display 800based upon the above-described display 700 and exemplary informationprovided therein. Thus, similar, exemplary display elements have beensimilarly numbered. The camera array window 740 in display 700 has beentoggled to reveal a recent activity window 840 showing activity (e.g.maintenance, cargo, bridge or safety), date, time and duration.

B. Event Tracking

FIGS. 9-12 show respective displays 900, 1000, 1100 and 1200 that can beaccessed via the activity tracker side menu 902, which can allow theuser to track various activities with respect to multiple vessels. FIG.9 particularly shows a display 900 that allows tracking of cargooperations. It includes filter boxes to allow selection of a trackingdate range 910, the relevant vessel 912 and cargo terminal 914. Anactivity box 920 with vessel, terminal, time and duration is shown. Thetime duration for various vessel cargo activities is further shown in acomparative bar graph 930. Cargo procedure visual review settings 940are also provided. Durations of specific cargo procedures (e.g. gangwayoperated, hose connected, etc.) are shown in segments window 950. Visualevents relevant to a cargo activity can be displayed in the window 960.A bar graph (histogram) 970 also shows comparative time durations over aseries of days or weeks.

FIG. 10 shows a maintenance activities tracking display 1000. Itincludes a filter 1010 for time and map 1012 showing a representation ofthe selected vessel and locations thereon being tracked. An activitywindow 1020 shows the area of interest as well as the time start, endand duration. A time log 1030 shows comparative area activities overtime as a series of bars. The activity location is also shown on a shiprepresentation in window 1040. A visual activity segment window 1050 isprovided and allows selection of visual maintenance events for displayin window 1060.

FIG. 11 shows a bridge behavior/activity tracking display 1100 with anassociated date range and vessel name filter 1110. A chart 1112, whichcan be shaded or color-coded, based upon level of activity is shown fora given date range and time block. The activity can be selected forpersonnel count or personnel movement based upon a selection switch1114. A map 1120 showing the vessel's bridge floorplan/layout isdepicted with a heat map that is color-coded or shaded based upon levelof motion or personnel density (as selected above). This heat map can beprojected into a three dimensional image from a particular bridge-basedsensor/camera as shown in windows 1130 and 1132. An activity segmentwindow 1140 is also provided, which allows selection of a visual eventdisplayed in window 1150. A particular sensor/camera can be selected fordisplay with switch 1152.

FIG. 12 shows a safety tracking display 1200 with associated time filter1210 as vessel map that allows selection of safety rounds and points ofinterest in specific areas, such as the upper deck (and port side,starboard side, etc.) or engine room. An activity window 1220 providesarea, safety round path, time and duration information. A bar graph 1230shows activities in selected areas (e.g. upper deck and engine room)over time. A visual review 1240 depicts a map of the vessel's selectedfloorplan (e.g. upper deck) and sub-area (e.g. port side, starboardside) with a map of rounds and associated sensor/camera locations. Asegment window 1250 also allows selection of visual events for displayon the window 1260.

C. Reports

FIGS. 13-16 show respective displays for reports related to cargoprocedures 1300, maintenance activities 1400, bridge behavior/activities1500 and safety 1600. A side menu 1302 allows for selection of reports.Such reports can be generated to provide subsequent data to the user andmanagement so that baselines can be established and decisions onpersonnel management, rules and operations can be made. Referring to thedisplay 1300 in FIG. 13, a selection box allows filtering based upon aparticular terminal locations 1310, date range 1312 and vessels 1314.Overall activity 1320 is listed with associated alarms, warnings and allclear statistics. Overall performance 1322 is also listed with anassociated bar graph. A plot graph 1324 and histograms 1326, 1328 canalso be displayed to chart various performance parameters as shown. Ahistoric baseline chart of performance is also displayed in window 1330.A comparative chart (shaded or color coded) of two or more vessels inthe fleet is depicted in window 140. The historic baseline can also bedisplayed as a pie chart 1350. Note that any acceptable graphic ordisplay of statistics can be provided in further embodiments for thisand other reports herein provided that the appropriate statistics andinformation to generate such a report exist. These reports can begenerated using appropriate computational techniques that should beclear to those of skill.

FIG. 14 shows a display 1400 for a maintenance activities report with afilter for date range 1410, vessel 1412 and location (e.g. upper deck,engine room, etc.) 1414. A map 1416 of the associated vessel deck isshown with deck highlights 1418 of areas of interest. An overallactivity box 1420 reports alarms, warnings and all clear conditions. Anoverall performance box 1422 includes a bar indicating level ofperformance. A bar graph (histogram) 1424 shows performance per vessel.An activity performance window 1440 plots hourly maintenance versus daterange using a color-coded or shaded scale to indicate level of activity.

FIG. 15 shows a display 1500 for generating reports on bridgebehavior/activity, which includes a filter for date range 1510 and atleast one vessel 1512. The display provides a window showing bridgeactivity sequences with alarms, warnings and all clear conditions. Awindow 1522 with overall performance, based upon an indicator bar isalso provided. A table 1524 can show comparative performance betweenvessels in a fleet and a plot 1526 of performance over time can beprovided. Additionally, performance for each vessel currently andhistorically can be provided in bar graphs (histograms) 1530. Acomparative plot 1540 of bridge activity versus time for a plurality offleet vessels can be also be depicted.

FIG. 16 shows a display 1600 for generating safety reports withassociated filter based upon date range 1610, vessel 1612 and reporttype 1614 (for example, safety rounds, PPE usage, etc.). An overallactivity window 1620 (e.g. over seven days) lists alarms, warnings andall clear conditions. An overall performance window with bar indicator1622 is also provided. Baseline safety activity in groupings of minutesis shown in bar graph (histogram). Historical performance betweenvessels in a fleet is also shown in a bar graph 1632. A plot showingsafety performance per hour for a range of dates is provided in window1640 with activity color coded or shaded based upon level.

V. Conclusion

It should be clear that the above-described system and methodeffectively provides a user with meaningful information on the statusand performance of various activities that are critical to safe andefficient operation of a vessel. This information includes relevantvisual information that can be highly beneficial in understandingactivities and providing evidence for follow-on analysis, etc. Thesystem and method can operate to deliver meaningful visual and otherinformation in near real time over small bandwidth communicationchannels. A graphical user interface can be provided to control thesystem and method, setting priorities, tracking and reporting activitiesas appropriate. The system and method can also be adapted to operateover larger bandwidth channels and/or with other types of remotelocations—for example islands, subterranean/underground, arctic andAntarctic stations, space-based locations, etc.

The foregoing has been a detailed description of illustrativeembodiments of the invention. Various modifications and additions can bemade without departing from the spirit and scope of this invention.Features of each of the various embodiments described above may becombined with features of other described embodiments as appropriate inorder to provide a multiplicity of feature combinations in associatednew embodiments. Furthermore, while the foregoing describes a number ofseparate embodiments of the apparatus and method of the presentinvention, what has been described herein is merely illustrative of theapplication of the principles of the present invention. For example, asused herein, the terms “process” and/or “processor” should be takenbroadly to include a variety of electronic hardware and/orsoftware-based functions and components (and can alternatively be termedfunctional “modules” or “elements”). Moreover, a depicted process orprocessor can be combined with other processes and/or processors ordivided into various sub-processes or processors. Such sub-processesand/or sub-processors can be variously combined according to embodimentsherein. Likewise, it is expressly contemplated that any function,process and/or processor herein can be implemented using electronichardware, software consisting of a non-transitory computer-readablemedium of program instructions, or a combination of hardware andsoftware. Additionally, as used herein various directional anddispositional terms such as “vertical”, “horizontal”, “up”, “down”,“bottom”, “top”, “side”, “front”, “rear”, “left”, “right”, and the like,are used only as relative conventions and not as absolutedirections/dispositions with respect to a fixed coordinate space, suchas the acting direction of gravity. Additionally, where the term“substantially” or “approximately” is employed with respect to a givenmeasurement, value or characteristic, it refers to a quantity that iswithin a normal operating range to achieve desired results, but thatincludes some variability due to inherent inaccuracy and error withinthe allowed tolerances of the system (e.g. 1-5 percent). Accordingly,this description is meant to be taken only by way of example, and not tootherwise limit the scope of this invention.

What is claimed is:
 1. A system for storage and bandwidth reduction ofvisual events in association with an activity comprising: at least onevisual event detector having at least one camera that acquires images ofthe activity, at least one processor receiving the images and a visualevent detection process operating at least in part on the processor thatprocesses the images; a queuing process that identifies the visualevents from the at least one visual event detector and stores the visualevents; and a communications process that enables a reduced-bandwidthcommunications channel adapted to transmit the stored visual events in areduced-bandwidth and queued format to a remote location, wherein thevisual events comprise activities related to at least one of bridgeoperations, cargo operations, maintenance operations, safety operations,and vetting and security with respect to a vessel, and wherein thevisual events are based upon software-based and hardware-based eventdetectors that identify predetermined characteristics within one or moreof the images and determine a state based upon the characteristics. 2.The system as set forth in claim 1 wherein the visual events providedata primitives that are derived into user information, the dataprimitives including at least one of statistics, pattern matches,metrics and time.
 3. The system as set forth in claim 2 wherein thestored visual events are communicated across the reduced-bandwidthcommunications channel based upon user-set priorities, wherein higherpriority events are communicated first in a queue from the stored visualevents.
 4. The system as set forth in claim 3 further comprising a userinterface that displays information of the at least one of the bridgeoperations, cargo operations, maintenance operations, safety operations,and vetting and security at a base location in one or more predeterminedpresentation formats.
 5. The system as set forth in claim 4 wherein theuser interface includes inputs that enable the user-set priorities. 6.The system as set forth in claim 4 wherein the user-interface includesinputs that enable adding or setting configuration parameters for thesoftware-based and hardware-based event detectors.
 7. The system as setforth in claim 4 wherein the user information includes at least one offleet reports on a plurality of vessels associated with the userinterface, vessel reports, occurrence of discrete events and occurrenceof complex events comprising a plurality of discrete events.
 8. Thesystem as set forth in claim 7 wherein the information is displayed in adashboard format comprising one or more visual windows, alerts andnumerical data.
 9. A user interface operated by a base locationprocessor located at a base location and receiving visual events over acommunication channel from a remote, moving or stationary object, inwhich the object includes at least one visual event detector having atleast one camera that acquires images of an activity, at least oneobject processor receiving the images and a visual event detectionprocess operating at least in part on the object processor thatprocesses the images and a communications process that enables thecommunication channel adapted to transmit stored visual events in queuedformat to the base location, the user interface comprising: a dashboarddisplay for displaying information associated with the object andenables configurations of at least one of (a) priority of visual eventsto be communicated over the communications channel and (b) configurationof characteristics associated with the visual events, wherein the visualevents comprise activities related to at least one of bridge operations,cargo operations, maintenance operations, safety operations, and vettingand security with respect to a vessel, and wherein the visual events arebased upon software-based and hardware-based event detectors thatidentify predetermined characteristics within one or more of the imagesand determine a state based upon the characteristics.
 10. The userinterface as set forth in claim 9 wherein the communications channel isa reduced-bandwidth, wireless communications channel.
 11. The userinterface as set forth in claim 10 wherein the object is a sea-goingvessel and the visual events relate to at least one of safety,maintenance, cargo handling, bridge operations, and vetting and securityrelative to the sea-going vessel.
 12. The user interface as set forth inclaim 11, wherein the dashboard display on the user interface providesat least one of configuration, status and performance informationrelative to the sea-going vessel.
 13. The user interface as set forth inclaim 12 wherein the dashboard display on the user interface provides atleast one of configuration, status and performance information relativeeach vessel of a fleet of vessels.
 14. The user interface as set forthin claim 11 wherein the information includes one or more forms ofdisplayed sensor data having at least one of a label, highlight, commentand annotation visually appended thereto.
 15. The user interface as setforth in claim 11 wherein the information includes a workflow providedbetween one or more users or vessels.
 16. The user interface as setforth in claim 11 wherein the communication channel is arranged totransmit control data from the user interface to one or more vessels andreceive feedback via the user interface relative to actions on the oneor more vessels with respect to the control data.
 17. A method forstorage and bandwidth reduction of visual events in association with anactivity comprising the steps of: acquiring images of the activity withat least one camera and performing visual event detection with aprocessor; identifying the visual events from software-based andhardware-based event detectors that identify predeterminedcharacteristics within one or more of the images and determine a statebased upon the characteristics and storing the visual events in a queuebased upon predetermined parameters, wherein the visual events compriseactivities related to at least one of bridge operations, cargooperations, maintenance operations, safety operations, and vetting andsecurity with respect to a vessel; and communicating with a remotelocation over a reduced-bandwidth communications channel adapted totransmit the stored visual events in a reduced-bandwidth and queuedformat to the remote location.